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DETAILED ACTION 

Response to Arguments 

1 . Applicant's arguments filed 5/1/2008 have been fully considered but they are not 
persuasive. Nishimura et al. (USPN 6778958) fully anticipate the limitations regarding 
"formatting a recognition result based on an identification of a non-verbalized 
punctuation mark after the non-verbalized punctuation mark has been inserted into the 
recognition result' in that the process of automatically inserting non-verbalized 
punctuation marks (e.g. commas and periods) into the recognition results is considered 
the same as formatting because the recognition results are now modified into structured 
sentences and clauses as opposed to fragmented individual words before the insertion 
(refer to col. 5, lines 30-32 and lines 57-67). Nishimura et al. also disclose the limitation 
regarding "the non-verbalized punctuation mark is predicted using at least one text 
feature and at least one acoustic feature that includes one or more of a length of period 
silence (col. 5, lines 35-67, using "pause" or "silence" and recognized words to 
determine punctuation marks; particularly lines 17-23 relying on acoustic model to 
determine punctuation mark; and lines 57-67 relying on language models (text) to 
determine punctuation marks). The only feature that Nishimura et al. lack is the 
"acoustic feature [is] a function of pitch of words near the period of silence". However, 
Rorex (US 7089184) teach that the acoustic feature is a function of pitch of words near 
the period of silence (col. 10, lines 17-27, tonality or pitch information is used to 
determine punctuation marks). 
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2. In response to applicant's argument regarding "Franz never suggests that 
punctuation is listed in the utterance hypotheses were non-verbalized" and the "non- 
verbalized punctuation mark is corrected" (page 6 of the REMARKS section), these two 
features are not claimed in claim 10. There is no clear indication in the claim 10 that a 
list of hypotheses must include punctuation marks. Specifically, claim 10 only recites 
"selecting a portion of the recognition result to be corrected that includes the non- 
verbalized punctuation mark". In other words, the selected portion includes punctuation 
mark. And Franz teaches this feature (figure 13, element 1304 includes a "period", and 
this period is not uttered by the speaker). Also, there is no clear indication in the claim 
that the "non-verbalized punctuation mark" must be corrected as argued. Rather, the 
claim specifically recites, "correcting the portion of the recognition result that includes 
the non-verbalized punctuation mark with one of a number of correction choices". Franz 
fully anticipates this limitation in that a list of hypotheses is presented for user's 
selection (figure 13). 

3. In response to applicant's argument regarding Rorex not being combinable with 
Nishimura, Rorex is only relied upon for the teaching of "acoustic feature includes a 
function of pitch of words near the period of silence" and nothing else. 

4. Previous office action has been withdrawn in favor of a new non-final office 
action due to examiner's use of a new reference for claim 1 1 . 
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Claim Rejections - 35 USC § 103 

5. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

6. Claims 1, 5-6, 13, 21, and 24-26 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Nishimura et al. (US 6778958) in view of Rorex (US 7089184). 

7. Regarding claims 1 and 13, Nishimura et al. disclose a method and apparatus 
(figure 2) of recognizing punctuation in computer-implemented speech recognition, the 
method comprising: 

performing speech recognition on an utterance to produce a recognition result for 
the utterance (col. 5, lines 5-34 or figure 2); 

identifying a non-verbalized punctuation mark in a recognition result {col. 5, lines 
5-34 or figure 2) including predicting the non-verbalized punctuation mark using at least 
one text feature and at least one acoustic feature related to the utterance {col. 5, lines 
35-67, using "pause" or "silence" and recognized words to determine punctuation 
marks; particularly lines 1 7-23 relying on acoustic model to determine punctuation mark; 
and lines 57-67 relying on language models (text) to determine punctuation marks); 

inserting the non-verbalized punctuation mark into the recognition result (col. 5, 
lines 48-67, inserting punctuation mark into the recognition result); and 



Application/Control Number: 10/718,042 Page 5 

Art Unit: 2626 

formatting the recognition result based on the identification of the non-verbalized 
punctuation mark after the non-verbalized punctuation mark has been inserted in the 
recognition result (refer to col. 5, lines 30-32 and lines 57-67; the recognition results is 
considered the same as formatting because the recognition results are now modified 
into structured sentences and clauses as opposed to fragmented individual words 
before the insertion); 

wherein the acoustic feature includes one or more of a length of a period of 
silence (co/. 5, lines 35-67, using "pause" or "silence" and recognized words to 
determine punctuation marks). 

Nishimura et al. fail to specifically disclose that the acoustic feature include a 
function of pitch of words near the period of silence. However, Rorex (US 70891 84) 
teach that the acoustic feature includes a function of pitch of words near the period of 
silence (col. 10, lines 17-27, tonality or pitch information is used to determine 
punctuation marks). 

Since Nishimura et al. and Rorex are analogous art because they are from the 
same field of endeavor, it would have been obvious to one of ordinary skill in the art at 
the time of invention to modify Nishimura et al. by incorporating the teaching of Rorex in 
order to improve non-verbalized punctuation recognition accuracy. 

8. Regarding claims 21 and 26, Nishimura et al. disclose a method and apparatus 
(figure 2) of recognizing punctuation in computer-implemented speech recognition 
dictation, the method comprising: 
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performing speech recognition on an utterance to produce a recognition result for 
the utterance {col. 5, lines 5-34 or figure 2); 

identifying a non-verbalized punctuation mark in a recognition result (col. 5, lines 
5-34 or figure 2); and 

determining where to insert the non-verbalized punctuation mark within the 
recognition result based on the identification using at least one text feature and at least 
one acoustic feature related to the utterance to predict where to insert the non- 
verbalized punctuation mark (col. 5, lines 5-34 or figure 2, using pause/silence to 
determine punctuation marks); and 

inserting the non-verbalized punctuation mark into the recognition result (col. 5, 
lines 48-67, inserting punctuation mark into the recognition result); and 

wherein the acoustic feature includes one or more of a length of a period of 
silence (col. 5, lines 35-67, using "pause" or "silence" and recognized words to 
determine punctuation marks). 

Nishimura et al. fail to specifically disclose that the acoustic feature include a 
function of pitch of words near the period of silence. However, Rorex (US 7089184) 
teach that the acoustic feature includes a function of pitch of words near the period of 
silence (col. 10, lines 17-27, tonality or pitch information is used to determine 
punctuation marks). 

Since Nishimura et al. and Rorex are analogous art because they are from the 
same field of endeavor, it would have been obvious to one of ordinary skill in the art at 
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the time of invention to modify Nishimura et al. by incorporating the teaching of Rorex in 
order to improve non-verbalized punctuation recognition accuracy. 

9. Regarding claims 5-6 and 24-25, Nishimura et al. fail to specifically disclose the 
method as in claims 1 and 21 , respectively, wherein the acoustic feature includes an 
average pitch of words near the period of silence, and wherein the acoustic feature 
includes a ratio of an average pitch of words near the period of silence. However, 
Rorex teaches wherein the acoustic feature includes an average pitch of words near the 
period of silence, and wherein the acoustic feature includes a ratio of an average pitch 
of words near the period of silence (col. 10, lines 17-27, tonality or pitch information is 
used to determine punctuation marks). 

Since Nishimura et al. and Rorex are analogous art because they are from the 
same field of endeavors, it would have been obvious to one of ordinary skill in the art at 
the time of invention to modify Nishimura et al. by incorporating the teaching of Rorex in 
order to improve punctuation recognition accuracy. 

10. Claims 7-9 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Nishimura et al. (US 6778958) in view of Rorex (US 7089184), and further in view of 
Kolster (US 5920877). 

1 1 . Regarding claims 7-9, Nishimura et al. fail to specifically disclose the method as 
in claim 1 wherein formatting the recognition result includes controlling or altering 
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spacing relative to the non-verbalized punctuation mark, wherein formatting the 
recognition result includes controlling or altering capitalization of words relative to the 
non-verbalized punctuation mark, and wherein the non-verbalized punctuation mark 
includes a period, and formatting the recognition result includes inserting an extra space 
after the period and capitalizing a next word following the period. However, Kolster 
teaches wherein formatting the recognition result includes controlling or altering spacing 
relative to the non-verbalized punctuation mark (col. 11, lines 38-50), wherein formatting 
the recognition result includes controlling or altering capitalization of words relative to 
the non-verbalized punctuation mark (col. 11, lines 38-50), and wherein the non- 
verbalized punctuation mark includes a period, and formatting the recognition result 
includes inserting an extra space after the period and capitalizing a next word following 
the period (col. 11, lines 38-50). 

Since Nishimura et al. and Kolster are analogous art because they are from the 
same field of endeavors, it would have been obvious to one of ordinary skill in the art at 
the time of invention to modify Nishimura et al. by incorporating the teaching of Kolster 
in order to put the received text into organized structure to make good presentation. 

12. Claims 10 and 12 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Nishimura et al. (US 6778958) in view of Franz et al. (US 6356865). 

1 3. Regarding claim 1 0, Nishimura et al. disclose a method of recognizing 
punctuation in a computer-implemented speech recognition, the method comprising: 
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performing speech recognition on an utterance to produce a recognition result for 
the utterance {col. 5, lines 5-34 or figure 2); 

identifying a non-verbalized punctuation mark in a recognition result (col. 5, lines 
5-34 or figure 2); 

formatting the recognition result based on the identification (refer to col. 5, lines 
30-32 and lines 57-67; the recognition results is considered the same as formatting 
because the recognition results are now modified into structured sentences and clauses 
as opposed to fragmented individual words before the insertion). 

Nishimura et al. fail to specifically disclose selecting a portion of the recognition 
result to be corrected that includes the non-verbalized punctuation mark; and correcting 
the portion of the recognition result that includes the non-verbalized punctuation mark 
with one of a number of correction choices. However, Franz et al. teach selecting a 
portion of the recognition result to be corrected that includes the non-verbalized 
punctuation mark (the operation of figure 13; the non-verbalized punctuation is the 
silence or pause portion taught in Nishimura reference; the selecting in Franz include 
silence portion between two words in figure 13); and correcting the portion of the 
recognition result that includes the non-verbalized punctuation mark with one of a 
number of correction choices (the operation of figure 13). 

Since Nishimura et al. and Franz et al. are analogous art because they are from 
the same field of endeavors, it would have been obvious to one of ordinary skill in the 
art at the time of invention to modify Nishimura et al. by incorporating the teaching of 
Franz et al. in order to speech recognition accuracy. 
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14. Regarding claim 12, Nishimura et al. fail to specifically disclose the method as in 
claim 10 wherein at least one of the correction choices does not include the non- 
verbalized punctuation mark. However, Franz et al. teach at least one of the correction 
choices does not include the non-verbalized punctuation mark (figure 13). 

Since Nishimura et al. and Franz et al. are analogous art because they are from 
the same field of endeavors, it would have been obvious to one of ordinary skill in the 
art at the time of invention to modify Nishimura et al. by incorporating the teaching of 
Franz et al. in order to speech recognition accuracy by enabling users to correct 
misrecognized words. 

15. Claim 11 is rejected under 35 U.S.C. 103(a) as being unpatentable over 
Nishimura et al. (US 6778958) in view of Franz et al. (US 6356865), and further in view 
of Roth etal.(US 7313526). 

16. Regarding claim 1 1 , the modified Nishimura et al. fail to specifically disclose the 
method as in claim 10 wherein at least one of the correction choices includes a change 
to the non-verbalized punctuation mark. However, Roth et al. further teach that at least 
one of the correction choices includes a change to the non-verbalized punctuation mark 
{figures 33-34, elements 3302-3304 and 1124 and 3418; selecting type of punctuation). 

Since the modified Nishimura et al. and Roth et al. are analogous art because 
they are from the same field of endeavors, it would have been obvious to one of 
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ordinary skill in the art at the time of invention to modify Nishimura et al. by 
incorporating the teaching of Roth et al. in order to speech recognition accuracy by 
enabling users to correct misrecognized words. 

Conclusion 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to HUYEN X. VO whose telephone number is (571)272- 
7631 . The examiner can normally be reached on M-F, 9-5:30. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Patrick Edouard can be reached on 571-272-7603. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 

/Huyen X Vo/ 8/28/2008 
Primary Examiner, Art Unit 2626 



Application/Control Number: 10/718,042 Page 12 

Art Unit: 2626 



